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This document presents a study related to teacher 
education programs in general, but with specific attention directed 
to, National Science Foundation (NSF) institute programs for 
in-service teachers* The study was designed to answer questions 
pertinent to an assessment instrument designed and used with the Ball 
State Oniversity NSF institute programs. The reliability of the 
instrument, the extent of participant-perceived change in classroom 
emphasis given to 57 instructional topics included in the insi ument, 
and whether a significant difference existed between the shift of 
emphasis of the 1972 institute participants and that of the 1973 
participants were determined. All of the 1973 members took part in a 
pre-institute assessment and in two post-institute follow-ups. One 
follow-up test was administered on the last day of the institute and 
one the following spring. Changes in instructional emphasis were 
evaluated by the sign test. Comparison of mean growth increments was 
accomplished using a t-test. Analyses showed that 
participant-perceived increases in the level of emphasis were 
significant. No significant differences were found between the mean 
growth increments of the two groups. Instrument reliability was 
established, using analysis of variance in a modified intraclass 
correlation formula, as .93 and .97 for the 1972 and 1973 groups, 
respectively. (Author/EB) 
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For nearly two decades, the National Science Foundation funded in- 
stitute programs that brought secondary school teachers of science and 
mathematics back to the college or university campus for further study. 
These institute programs began at the University of Washington In 195A, 
and by 1965 numbered 449 nation-wide. |^ll] Many of these institutes 
were designed to update the basic subject matter competence of teachers 
and to familiarize them with, and prepare them to teach, the courses pro- 
duced by the various science curricular studies. The major intent of most 
institutes was to update secondary school science and mathematics teachers 
in both science content and instructional methodology. 

As NSF policy changed in the early 1970s and as financial support for 
Institute programs began to decline, institute directors and other concerned 
educators began to search for mechanisms to assess the effectiveness of in- 
stitute programs. Numerous articles summarize studies which were designed 
to assess the effectiveness of NSF institutes, [l, 3, 7, 8, 9, 10, 12, uj 
Evidence from these studies suggests that NSF-funded institutes have been 
influential in improving teacher competence and in fulfilling the objectives 
set forth by NSF. However, most of this evidence Is of a subjective nature, 
usually obtained through the use of a questionnaire In which the participants 
simply responded to questions that pertained to how, or how much, the in- 
stitute helped them. 

The biology institute faculty at Ball State University has been of the 
opinion for some time that an Institute evaluation instrument was needed that 
would provide data of a nature that could be analyzed statistically. A first 
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step pertaining to statistical analysis of data concerning teacher per- 
ceptions of institute effectiveness was taken by Hendren, Mertens and 
Nisbetf [s] This study was initiated in the spring of 1972 when a pre- 
institute assessment of the level of emphasis given to each of 55 in- 
structional topics was made by each teacher selected to participate in the 
1972 summer institute at Ball State University. This was followed by a po3t« 
institute assessment of the same 55 instructional topics and a determination 
of the amount of participant growth that had occurred with respect to these 
topics. The data collected through the pre-institute assessment provided a 
baseline with which comparisons were made concerning teaching emphasis given 
these topics after the institute. The establishment of these baseline data 
made a follow-up study amenable to statistical analysis. 

By way of contrast i most institutes have been evaluated by post-institute 
questionnaires only. Such studies produced data which were not easily sub- 
jected to statistical analysis because no baseline data were available for 
comparison. The study by Hendren et al. [ 8 J produced evidence of the change 
in emphasis given by participants to 55 instructional topics and did much to 
document institute effectiveness as perceived by the participants themselves. 
At least one serious doubt remained^ however: *'Was the assessment instrument 
reliable? Did the assessment instrument measure accurately what it was in- 
tended to measure?** 

Unanswered Questions . As prf^parations for the 1973 biology institute at 
Ball State University were being finall^ed^ the decision was made to pre/post 
assess the participants in a manner similar to that used with the 1972 in- 
stitute participants^ The assessment instrument was modified slightly 
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(e,g«, two additional items were added, making e. total of 57), and this new 
version was administered to the participants in the 1973 institute. The 
study summarized here was designed to answer three questions that are per- 
tinent to the assessment instrument and to the Ball State University NSP 
institute programs. 

The first concern of those administering the assessment instrument was 
the answer to the question, "Is the assessment instrument accurately measur- 
ing what it was intended to measure? Does the instrument effectively communicate 
to the participants so that valid interpretations of assessment results can be 
made?" Determination of the reliability of the assessment instrument became 
the primary task of this study. 

The second concern of this stady was to determine the extent of par- 
ticipant-perceived change in classroom emphasis given to the 57 instructional 
topics included in the assessment instrument. Specifically, the answer to 
the following question was sought: "Has statistically significant change in 
emphasis taken place following participation in the institute program?" 

The third goal of the study was to determine whether or not a statistically 
significant difference existed between the shift of emphasis of the 1972 in- 
stitute participants and that of the 1973 participants. The third major 
question to be answered was, "Did one group of participants change Its emph^^i^ls 
more or less than the other group as measured by the assessment instrument?" 
Methods , All of the 1973 institute participants took part In a pre-lnstltute 
assessment and a post-institute follow-up using a modified form of the 
assessment instrument employed in 1972, In April 1973, before the institute 
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began, each of the 40 participants was asked to assess the emphasis he/she 
had placed on each of 57 instructional topics "during academic year 1972-73. 
Th.'>se topics included relevant teaching methodology as well as current 
developments in biological science. Each participant assessed his current 
level of emphasis, his desired level of emphasis, and the significance of 
each of the 57 topics for his students on a scale of 1-7. (See Table 1 for 
interpretation of the assessment scale.) It was anticipated that "desired 
emphasis" and "significance for students" would show a strong positive 
correlation and perhaps serve as a cross check within the assessment in- 
strument. Surely, a teacher who thought that a particular topic was significant 
to students would desire to emphasize that topic in the classroom. A compar- 
ison of columns B and C in Table 2 reveals that the data generally confirm 
this prediction. The data collected through the pre-ins'titute assessment 
(Table 2, column A) provided a baseline with which comparisons may be made 
concerning teaching emphasis given these topics after the institute. 

The same assessment form was again administered to the participants on 
the last day of the summer institute program and the participant was asked 
to indicate the level of emphasis he/she desired to place on each of the 
instructional topics during 1973-74. Finally, the assessment form was mailed 
to each of the 40 participants in the spring of 1974, one year after the 
pre-institute assessment and seven months after the close of the summer 
institute. The same 57 topics were again assessed with respect to the level 
of emphasis actually given to the topic in the 1973-74 school year. Table 
2 summarizes the findings of this study. The data obtained from the 1973-74 
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administration of the assessment form and from the use of the similar form 
in 1972 [s'j provide the basis for the remainder of this article. 

Data and Discussion 
Reliability of the Assessment Instrument . Obviously the data obtained in a 
study such as has been described, are meaningful only if the assessment in- 
strument can be demonstrated to have a reasonable measure of reliability. 
"Reliability" in this sense is simply "how accurately [the device] measures 
whatever it does measure;, [u, p. I??! One approach to determining reliability 
is to administer the assessment instrument on several occasions to determine 
whether it performs similarly on repeated trials. Establishment of reliability 
by repeated performance is statistically valid when the resulting data con- 
stitute a ranking of alternatives (e.g., right or wrong responses). Clearly, 
responses on a sliding scale assessment instrument such as was used in this 
study, do not provide data of the requisite ranking type. 

In the present study, reliability must be established for ratings rather 
than for rankings . For rating data it is possible to determine internal 
reliability; i.e., reliability may be established by using an analysis of 
variance in a modified intraclass correlation formula. [^4, 6^ In its simplest 
form this involves an analysis of variance and the calculation of the re- 
liability coefficient, r, for all raters (participants completing the assess- 
ment), where 

J, ^ mean square (all items) - mean square (error) 
mean square (all items) 

The value of r may range from -1.0 to 1.0 with the values between 0 and 1.0 
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indicating positive correlations between the raters* mean ratings. 

Table 3 summarizes the analysis of variance statisUcs and reliability 
coefficents calculated for the post-institute assessment results for both 
the 1972 and 1973 participants. For the 1972 participants, for example, 
30 raters (participants) provided complete data for the 55 items assessed by 
the instrument. In the case of the 1973 participants 33 raters each assessed 
the same 55 items (the two new items - numbers 18 and 27 in Table 2 were not 
included in the analysis). Using the data for the 1973 participants as an 
illustration, it may be inferred that if the 33 ratings for each of the 55 
items were averaged and if we could correlate these averages with a similar 
3et of averages from a comparable group of participants, the result would be 
about 0.97. Thus, the extremely high reliability coefficients, .93 and .97, 
provide positive evidence that the assessment instrument is truly "measuring 
accurately what it is intended to measure." The data obtained as a result 
of administering this instrument may be interpreted with a great deal of 
confidence, since a reliability of this magnitude exists. 
Participant growth in 1973-74. The second goal of this study was the evalu- 
ation and interpretation of data obtained by the use of the assessment .ln« 
strument with the 1973 institute participants. With the reliability of the 
instrument clearly established, one can have confidence in the data obtained 
by the administration of the instrument. 

The mean assessment values reported in Table 2 suggest that, as perceived 
by the participants, the institute has been effective in stimulating an increase 
in the level of emphasis given to the 57 instructional topics. The data for 
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each of the topics were analyzed using a test designed to determine whether 
the increase in the level of emphasis was, in fact, statistically significant. 
The test used for this purpose, called the sign test, "is based on the signs 
of the differences (whether they are positive or negative) ignoring their 
magnitudes" [s, p. 295] 

An example of hcw the data were subjected to the sign test follows: 
For item 27, "societal problems resulting from over population and mis-use 
of technology," 24 of the participants increased the level of emphasis in 
their teaching during academic year 1973-74, 3 decreased their emphasis, 7 
did not change, and the data were incomplete for 2 of the participants. 
Occasional omissions on the completed assessment forms account for instances 
of incomplete data. In addition, four participants did not complete the 
post-institute assessment (spring 1974), thus reducing the total population 
studied to 36. Using the data for item 27, the null hypothesis, that the in- 
stitute had no effect on determining the level of instructional emphasis, was 
tested. This is equivalent to testing the hypothesis that a positive change 
and a negative change were equally probable; that is, the chance of getting 
an increased level of emphasis (positive change) "is £ » 0.50 against the 
one-sided alternative that £>0.50;' fs, p. 296] For this purpose the statistic 
£ was calculated as follows: 

X - £ 

2 = 

- o* 

where x « the number of positive changes « 24; £ « rj£, where n - total number 
of changes (both positive and negative) » 24 + 3 « 27, and £ » 0,50; and (T « 
yn£(l-£) » y27(0.50)(0.5O), because £ - l-£ « 0.50. 

Therefore, z 2^ -13,5, , « 10.5 « 4.04. 

V^(0750)"(0.50) y 6.75 
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Using a table of z values [^"J , significance at the 1% level may be 
determined as z = 2,33, Hence, in the case of this topic, •'societal problems 
resulting from over population and mis-use of technology," £ = A, 04 is 
statistically significant at the 17. level and the null hypothesis is rejected. 
Therefore, for this topic, it may be concluded that the participant-perceived 
change in the level of emphasis following the Institute is statistically 
significant. The reader will note that statistically significant increases 
(at the 17o level) in the degree of emphasis were given to 49 of the 57 in- 
structional topics. Increases in emphasis were statistically significant at 
the 57, level for an additional six topics (items 7, 16, 19, 44 51, and 54). 
For only two items (21 and 38) among the 57 topics was there no evidence of 
statistically significant, growth. 

Since the type of institute assessment described herein was also employed 
with the participants in the 1972 summer institute, this study affords the 
opportunity to compare the two groups of participants and to obtain some 
evidence of the reliability of the assessment instrument in yet another way. 
For example, a comparison of Table 2 in this study with the comparable table 
in the study by Hendren et al. 8 ] reveals statistically significant in- 
creases (at the 17, level) with respect to 40 topics for both groups of par- 
ticipants. The increase in emphasis with respect to item 7 for both groups 
was statistically significant at the 57. level. Since the 1973 assessment 
form included two items (18 and 27) not included in the 1972 assessment, the 
two groups of participants gave similar responses to 41 of 55 topics included 
in both assessment instruments. These comparisons further increase confidence 
in the reliability of this technique of participant self-assessment. 
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When Hendren et a I. [sj compared the "desired level of emphasis" be- 
fore the institute (spring 1972) with the "desired level of emphasis" on 
the last day of the institute, they found that for 40 of the 55 topics the 
desired level of emphasis decreased following participation in the Institute. 
This observation was attributed to a more realistic post-institute view on 
the part of the participating teachers as to what they would be able to 
accomplish upon returning to their respective classrooms. Such was not the 
case, however, for the 1973 participants (compare columns B and D in Table 
2). For 37 of the 57 instructional topics in the 1973 assessment instrument, 
the participants increased the desired level of emphasis after participating 
in the institute. This seems to suggest that the institute was a motivating 
force in increasing participant interest in tV?.3e institute topics. 
Comparison of Growth Increments for Tjjo Groups of Participants . The third 
goal of this study was to compare the participant-perceived chwuges in In- 
structional emphasis between the 1972 and the 1973 groups of institute par- 
ticipants. Several observations suggest that the differences. If any, will 
be slight. These observations include: (I) the goals and objectives of 
the two institutes were similar; (2) the participants in the two institutes 
were all biology and/or life science teachers at the secondary school (7th- 
I2th grades) levelj (3) the assessment instruments used for the two groups 
of participants were identical with respect to the 55 items Included in the 
analysis; (4) results of the sign tests, as stated above, indicate that growth 
was similar for many of the 55 Items assessed; and (5) the reliability of 
the instrument has been clearly established. 
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In order to compare the two groups statistically, a standard two- 
tailed t^ test was applied to the null hypothesis that ''no significant 
difference exists between the growth (change In instructional emphasis) of 
the two groups." In order to prepare the data for this test, the difference 
in the mean growth for each item from the time of pre«insti tute assessment 
until the time of post-institute assessment (actually seven months after the 
Institute, as explained above) was calculated for each group of participants. 
The mean of these 55 differences (the two new items in the 1973 assessment 
form were discounted) was then calculated for each group. All statistics 
relevant to the JL test are summarized in Table 4« The calculation of a 
value for Jt follows* Note that in both the explanation below and in Table 
4, "group 1" refers to the 1972 participants and "group 2" refers to the 1973 
participants* 

If - the mean for group 1 « 1.0836, » the mean for group 2 « 

r2 K 2 

x^ » the sum of squared deviations for group 1 » 13.3952, « 

the sum of squared deviations for group 2 « 14.5563, and n « 55 » the 

number of items assessed by each group, then according to Blommers and 

Lindquist [jZ j , 




t « 




Substituting the numerical values in this formula gives: 



1.0836 - 1.2545 



t - 



1 373952 4- 14.'556"3Y 7 l" . I 
55 + 55 - 2 ) I 55 



then, t = 



/ 55+55-2/55 ' 55 

-0.1709 = -0.1709 



727.9515 \ M ^1 V . 0094 U 2 

V 108 i 55 



finally, t = -1.7617. 

Interpretation of this value of t^ using the appropriate number of 
degrees of freedom (108), reveals that the null hypothesis, that there is 
no significant difference between the mean growth increments of the two 
groups of participants, may be accepted since the probability associated 
with t^ « -1.7617 is 8*09%. Thus, the difference (X^ " ^2 "O^^^OQ) between 
the mean growth increment (Xj l.0'^36) of group I and the mean growth in- 
crement (X^ 1.2545) of group 2 is not statistically significants This 
finding provides a type of check on, and reinforcement of, the other stat- 
istical tests reported earlier in this article. 

Summary 

The implications of this study for the assessment of teacher education 
programs in general, and NSF science education programs in particular, would 
appear to be quite significant. The instrument used in pre/post assessment 
of the instructional emphasis given by in-service biology te-^chers to 55 
topics yields data that are amenable to a number of kinds of statistical 
analysis • Instrument relUbility was established using an analysis of vari- 
ance in a modified intraclass correlation formula* Changes in instructional 
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emphasis with respect to each assessment item were evaluated by the sign 
test. Finally, comparison of mean growth increments for different groups 
of teachers was accomplished by using a t^ test. 

The fact that the assessment instrument was shown to be highly reliable 
(r ^ 0.93 for 1972 data and £ - 0.97 for 1973 data) lends credence to the 
sign test calculations, which revealed that in the case of the 1973 par- 
ticipants, for examp!.e, the participant-perceived increases in the level 
of emphasis given to 49 of 57 instructional topics were statistically 
significant at the 1% level. The reliability of the instrument is further 
reinforced, since comparable results were obtained with two separate groups 
of participants. A statistically significant difference between the mean 
growth increments of the two groups of participants, as determined by the 
t^ test, was not found to exist* 
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Table 1, Rating scale used to assess each of 57 instructional topics. 



1 « virtually no emphasis 

2 = slight emphasis 

3 - some emphasis, but below average 

4 « average emphasis 

5 » slightly above-average emphasis 

6 " considerable emphasis 

7 « high level of emphasis 

'-"""j^ ■ 
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TabU 2 . Mean assessment values, for 1973 NSF Institute participants. A, current emphasis, 
spring 1973 (before institute); II, desired emphasis, spring 1973 (before institute); C, 
perceived significance for students, spring 1973 (before institute); D, desired emphasis, 
August 1973 (end of institute); E, actual emphasis, spring 197A; F, sign test 2 value 
(** « significant at the 17. level; * « significant at the 5% level). 



INSTITUTE TOPIC 


A 


B 


C 


D 


E 


F 


Molecular biology 














1, Biologically significant molecules 


3.86 


4.89 


4.49 


4.80 


4.71 


2.75** 


2. Functional groups 


3.11 


3.91 


3.71 


4.11 


3.80 


2.56^ 


3. Chemical constituents of cells 


3.31 


4.34 


3.88 


4.72 


4.34 


3.27^ 


A. Chromatography 


1.9A 


3.44 


3.18 


4.25 


3.12 


3.54 


5. Electrophoresis 


0.53 


1.17 


1.11 


2.72 


1.34 


4.12 


6, Autoradiography 


0.43 


1.17 


1.21 


2.30 


1.53 


4.47 


Cell structure and function 














7. Cell organelles 


4.77 


5.36 


5.00 


5.26 


5.20 


2.13 


8, Cyclosis 


2.54 


3.22 


2.97 


4.37 


4.41 


•irk 

3.80 


9. Photosynthesis 


4.66 


5.46 


5.18 


5.69 


5.49 


•kit 

3.96 


10. Energy production 


4.03 


5.08 


4.89 


5.31 


5.34 


•kic 

4.43 


11. Mitosis 


4.46 


5.33 


4.89 


5.09 


5.34 


3.13 


12. Melosls 


4.46 


5.19 


5.00 


5.09 


5.31 


3.13 


Genetic biology 














13. Basic principles of Mendellan 
genetics 


4.09 


5.09 


5.00 


4.94 


5.00 


2.75 


14* Human sex chromatin 


3.11 


4.88 


4.71 


4.06 


4.14 


3.53** 


IS. Human chromosome aberrations 


3.15 


4.63 


4.63 


4.83 


4.61 


4.43 


16. Drosophila senetics 


2.29 


3.71 


3.76 


3.86 


3.32 


it 

1.96 


17. Sordaria genetics 


0.23 


0.76 


0.74 


3.17 


1.70 


5.00 



o 



Genetic biology 



18. Societal problems resulting from 
new genetics knowledge and tech- 
nology 


2.24 


4.86 


4.74 


5.34 


4.51 


4.85 


BlcioKic diversity 














19. Evidences for evolution 


3.54 


4.64 


4.37 


4.81 


4.51 


2.20* 


20t Mechanism of evolution 


3.23 


4.50 


4.11 


4.64 


4.37 


3.40 


21. F'rlnciples of biosystematics 


3.63 


4.31 


4.09 


4.22 


3.83 


0.63 


Kcoloj2;lcal principles and environmental 














problems 














22. Ecologic succession 


3.69 


4.81 


4.60 


4.86 


4.29 


2.65^ 


23. Competitive exclusion (Cause's 
principle) 

• 


1.58 


2.50 


2.43 


4.39 


3.12 


itit 

3.65 


24, Problems of pollution 


4. 31 


5.47 


5.54 


5.86 


5.04 


2 65 


25. KuLrophication and water quality 


3.06 


4.69 


4.76 


5.31 


4.23 


3.53 


26. Population growth curves 


3.31 


4.69 


4.74 


4.67 


4.56 


3.27** 


27. Societal problems resulting from 
over population and mis-use of 
technology 


3.26 


5.14 


5.29 


5.31 


4.89 


4.04 


Philosophic basis for biology 
instruction 














28. Assessing the direction and 
significance of developments 
in biology training 


• 

3.35 


4.94 


4.82 


4.63 


4.91 


• 


29. Evaluating teacher goals 


3.56 


5.03 


4.70 


5.57 


5.75 


4.95** 


30. Writing performance objectives 


3.56 


4.97 


4.91 


6.00 


5.58 


4.62** 


31. Considering the affective domain 


2.94 


4.27 


4.16 


5.57 


5.47 


4.38 


32. Considering the cognitive domain 


3.39 


4.18 


4.00 


5.54 


5.41 


3.78 


33, Evaluating the function of 
evaluation 


2.79 


4.09 


3.88 


5.03 


5.31 


4. 54 


34. Constructing test items 


4.06 


5.31 


5.15 


5.94 


5.81 


3.89** 
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INSTITUTK TOPIC 


A 


B 


C 


8HST 

D 


COPY j 

E 


F 


Interpersonal challenp;es for the 
biology teacher 














35. Considering characteristics of 
the effective teacher 


3.66 


5.33 


5.21 


5.78 


5.43 


2.89** 


36. Assessing the significance of 
self-concept 


3.47 


4.74 


4 70 
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37. Enlisting administrativ.e support 


3.91 


5.14 


5.18 


5 17 


S 3A 


A 90** 


38. Working with peers 


4.97 


6.00 


5.89 


6.08 


5.63 


1.63 


Curricular materials for biology 
Instruction 














39. Teaching BSCS standard courses 


3.00 


4.06 


3 91 


3 61 


3 Q1 


9 An 


40. Using BSCS lab blocks 


1.18 


2.62 


2.55 


3.89 


2.79 


3.53 


41. Working with second level BSCS 
materials 


0.62 


1.30 


1.27 


1.56 


1.97 


_ ^ . 7C7C 

3.96 


42. Working with BSCS special 
materials 


0.94 


2.08 


2.26 


2.92 


2.44 


4.81 


43. Developing teaching units for 
local use 


1.66 


3. 72 


3 83 




*♦ • vU 


A 97 


Teaching strategies 














44. Developing audiotutorial materials 


.2.46 


4.67 


4.71 


3.61 


3.36 


if 

1.73 


45. Developing electronic-response 
materials 


0.69 


1.97 


1 QA 


1 72 


1 11 


9 AA 


46. Teaching through inquiry 


3.97 


5.69 


5 57 


J # O 7 


^ LC\ 

J . ^u 




47. Assessing contract learning 


1.71 


3.71 


3*58 


3 97 


9 OA 


9 71 


48. Implementing modular scheduling 


1.45 






3 




o A1 


49. Teaching controversial topics 


3.03 


4*49 


4 56 




A AA 


A 97 


ju, experiencing microteaching 


1.29 


2.29 


1.91 


3.47 


2.35 


3.02 


51. Using TV In biology instruction 


1.62 


3.26 


3.33 


2.78 


2.38 


2.24 



UJ 




INSTITUTE TOPIC 


A 




C 


X) 


E 


F 


Facilities, materials and resources 
for bioloj^y instruction 














52. Guidelines for the biology library 


2.24 


4.b0 


4.53 


3.86 


3.74 


3.66** 


53. Selecting equipment and facilities 


3.79 


5.06 


5.06 


4.47 


5.03 


3.40 


54. Designing biology laboratories 


2.31 


3.66 


3.88 


3.17 


3.49 


1.96 


55, Identifying sources of supplies 
and living materials 


3.54 


5.22 


5.00 


4.83 


5.09 


2.65** 


56. Using outdoor education areas 


2.94 


5.61 


5.69 


5.14 


4.00 


2.45** 


57. Employing community resources 


2.57 


5.33 


5.37 


5.00 


4.14 


4.04 



7) 

'.a 
■ J 

». 

ERIC 



Table 3, Internal reliability of assessment instruments was calculated using data 
obtained from administering the assessment in the spring following each institute. 



A, Assessment form completed in spring 1973 by 1972 participants (30 countable) 

Sources Sums of squares degrees of freedom mean square 
of var iabi lity 

Items 1596,7991 54 29,5704 

Haters 1289.7384 29 44.4737 

Error 3195.8009 1566 2.0407 



Total 6082.3384 

reliability (all raters) 



1649 



29.5704-2.0407 
29.5704 



.9310 



B, Assessment form completed in spring 1974 by 1973 participants (33 countable) 

Sources Sums of squares degrees of freedom mean square 

of variability 

Items 3532.0313 54 65.4080 

Raters 1048.0400 32 32.7512 

Error 3865.9527 1728 2.2372 



Total 



8446,0240 



1814 



reliability (all raters) = 



65.4080-2.2372 
65.4080 



.9658 



BEST tm mmmit 



Table ^, Statistics obtained in analysis of data for calculating a ^ test for the 
comparison of the means of differences of pre/post institute assessment data for 
two groups of participants. The calculated value of t. ^-1. 76174 The two-tailed 
probability level with 108 degrees of freedom =5 8.097o. 



Component of calculation 


Vjl. v.'U p -i- 

(1972 participants) 


fit* nun 2 

(1973 participants) 


Mean of differences of 55 






pre/post assessed items 


1.0836 


1.2545 


Variance 


.2435 


.2646 


Standard deviation 


.4935 


.5144 


n (number of assessment items) 


55.0000 


55.0000 


Sum (jf scores 


59.6000 


69.0000 


Sum of scores squared 


77.9800 


101.1200 


Sum of squared deviations 


13.3952 


14.5563 



Li 
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